1 link tagged with all of: context window + backburner + llama.cpp + split inference
Click any tag below to further narrow down your results
Links
Backburner splits a 27-billion-parameter language model between a MacBook Pro and iPhone connected via USB-C, achieving 29-44% faster inference on long contexts by distributing computation across both devices' GPUs. The system maintains identical outputs to running the model on the Mac alone while supporting up to 196k tokens of context on an iPhone 17 Pro Max.
- Split prefill workload: Mac runs layers 1-40 while iPhone runs layers 41-64 in parallel, with the iPhone holding older KV cache pages past 64k context and computing attention over them during both prefill and decoding.
- Measured performance gains: 44% faster throughput at 16k context (157 vs 109 tok/s), and ability to maintain 8-bit precision at 128k tokens versus the Mac alone dropping to 4-bit to fit in memory.
- Greedy decoding is token-identical between Mac-only and Mac+iPhone configurations across tested context lengths (8k, 32k, 140k), confirming the split computation produces correct results.